%% fluidity.tex
%%
%% Created 2010-08-01 by the Fluidity in InfoVis team (Dagstuhl).

%% ---------------------------------------------------------------------
%% Document class
%% ---------------------------------------------------------------------
\documentclass{article}

%% ---------------------------------------------------------------------
%% Packages 
%% ---------------------------------------------------------------------

% Times fonts
\usepackage{times}

% Advanced citations package
%%\usepackage{cite}

% Permits inclusion of PNG, JPG, and PDF files under pdflatex
\usepackage{graphicx}
%%\usepackage{epstopdf}

%% For URLs
\usepackage{url}

%% ---------------------------------------------------------------------
%% LaTeX Setup
%% ---------------------------------------------------------------------

% Set up for double space
% \renewcommand{\baselinestretch}{2}

%% New commands and shortcuts
\newcommand{\papertitle}{Fluid Interaction for Information Visualization}
\newcommand{\shortpapertitle}{Fluid Interaction for InfoVis}

% Set up the margins and text size
\oddsidemargin 4.5pc
\evensidemargin 4.5pc
\advance\oddsidemargin by -1in  % Correct for LaTeX gratuitousness
\advance\evensidemargin by -1in % Correct for LaTeX gratuitousness
\marginparwidth 0pt             % Margin pars are not allowed.
\marginparsep 11pt              % Horizontal space between outer margin and
                                % marginal note
\topmargin 4.5pc                % Nominal distance from top of page to top of
                                % box containing running head.
\advance\topmargin by -1in      % Correct for LaTeX gratuitousness
\textheight 666pt       	% 9 1/4 column height
\textwidth 42pc         	% Width of text line.

%% ---------------------------------------------------------------------
%% Title, author(s), and affiliation
%% ---------------------------------------------------------------------

%% IVS format instructions from: 
%% http://www.palgrave-journals.com/ivs/author_instructions.html

% Paper title
\title{\papertitle}

% Authors
\author{
  Niklas Elmqvist\footnote{School of Electrical
    \& Computer Engineering, Purdue University, 465 Northwestern Ave,
    West Lafayette, IN 47907, USA, E-mail: elm@purdue.edu.}
  \and Andrew Vande Moere 
  \and Hans-Christian Jetter
  \and Daniel Cernea
  \and Harald Reiterer
  \and T. J. Jankun-Kelly
}

\date{}

%% ---------------------------------------------------------------------
%% Document Beginning
%% ---------------------------------------------------------------------
\begin{document}

% ----------------------------------------------------------------------
% Title & Abstract -- Should be on its own page
% ----------------------------------------------------------------------

\maketitle

% \begin{center}
%   {\LARGE{\papertitle}}
% \end{center}

\begin{abstract}
  Despite typically receiving little emphasis in visualization
  research, interaction in visualization is the catalyst for the
  user's dialogue with the data, and, ultimately, the user's actual
  understanding and insight into this data.
  There are many possible reasons for this skewed balance between the
  visual and interactive aspects of a visualization.
  One reason is that interaction is an intangible concept that is
  difficult to design, quantify, and evaluate.
  Unlike for visual design, there are few examples that show
  visualization practitioners and researchers how to best design the
  interaction for a new visualization.
  In this paper, we attempt to address this issue by collecting
  examples of visualizations with ``best-in-class'' interaction and
  using them to extract practical design guidelines for future
  designers and researchers.
  We call this concept \textit{fluid interaction}, and we propose a
  operational definition in terms of the direct manipulation and
  embodied interaction paradigms, the psychological concept of
  ``flow'', and Norman's gulfs of execution and evaluation.

  \bigskip
  
  % Up to six keywords below:
  \textbf{Keywords:} fluidity, flow, embodiment, design, information
  visualization, human-computer interaction.

\end{abstract}

%\newpage

% ----------------------------------------------------------------------
% Body
% ----------------------------------------------------------------------

\section{Introduction}

Not all information visualization (InfoVis) tools are created equal.
As all users of such tools know, while we can expect any competent
tool to be able to represent abstract data in graphical form, there is
a certain class of InfoVis tools that take this a step further through
engaging, compelling, and even absorbing user experiences that turn
the analytical sensemaking~\cite{Russell1993} process into a
pleasurable task.
However, the academic portion of the InfoVis field in general puts
little emphasis on design, aesthetic, and user experience aspects of
information visualization tools, and so far there has been virtually
no effort towards characterizing this class of InfoVis tool in the
research community.
In contrast, much of information visualization in the real world is
directly concerned with creating compelling---even playful---and
beautiful tools that are capable of capturing the attention of general
users on the Internet and in public spaces such as museums, exhibition
halls, and corporate lobbies.

In this article, we try to remedy this state of affairs by proposing a
unifying concept for both researchers and practitioners that captures
this class of \textit{InfoVis exemplars} in a single definition:
\textit{fluid interaction}.  
Fluidity in information visualization is an elusive and intangible
concept characterized by smooth, seamless, and powerful interaction;
responsive, interactive and rapidly-updated graphics; and careful,
conscientious, and comprehensive user experiences.
Our hypothesis is that an InfoVis tool that exhibits this fluidity in
all aspects will transform the sensemaking process into an efficient,
illuminating, and even enjoyable experience because it helps the user
stay in the flow~\cite{Csikszentmihalyi1991} of the work process.
However, creating a fluid design is far from trivial, mainly because
of this intangibility and elusiveness.

To better illustrate our definition of fluid interaction for
information visualization, we collect and describe a subset of these
exemplar InfoVis tools from the scientific community, including
BabyNameVoyager~\cite{Wattenberg2005},
Facet-Streams~\cite{Jetter2011}, and
Scatter/GraphDice~\cite{Bezerianos2010, Elmqvist2008a}.
Unlike most existing research articles in information visualization,
we also collect exemplars from outside the scientific community, such
as from the design, aesthetics, and infographics communities.
These ``real-world'' exemplars include m{\ae}ve~\cite{Bardzell2010,
  Maeve, Nagel2009}, We Feel Fine~\cite{WeFeelFine}, and the
interactive holographics from the film Iron Man 2~\cite{IronMan2}.

Using these exemplars as a starting point, we derive tips, guidelines,
and principles for how to achieve fluidity in InfoVis in terms of both
interaction and visual representations.
We hope that these snippets of practical experience, elevated almost
to the level of being design idioms and patterns~\cite{Gamma1995},
will help developers of InfoVis tools---academics and practitioners
alike---to build better, more rewarding, more captivating, and
ultimately more efficient information visualization tools that will
propel our field to make an even bigger impact in both our own
scientific community as well as the real world.

In the rest of this paper, we first survey the state of the art in
fluid interaction, design, and user experience for information
visualization (Section~\ref{sec:related-work}).
We then attempt to define the concept of fluid interaction through a
detailed list of properties that we think characterize fluid InfoVis
tools (Section~\ref{sec:definition}). 
These properties are all exhibited in the context of our InfoVis
exemplars, that we describe next (Section~\ref{sec:exemplars}).  
Starting from these exemplars, we derive design guidelines for fluid
information visualization (Section~\ref{sec:implications}), and
conclude the paper with our vision for future research directions in
design aspects of information visualization
(Section~\ref{sec:vision}).

\section{Related Work}
\label{sec:related-work}

Unlike its sister field of human-computer interaction, where design
and user experience are major components, InfoVis research articles
generally do not place much emphasis on interaction design aspects of
information visualization.
The few papers that discuss these topics tend to do so from a purely
scientific, engineering, or implementation viewpoint.
For example, although Amar et al.~\cite{Amar2005} present a meta-level
review of typical analysis t+asks and Yi et al.~\cite{Yi2007} collect
seven archetypes of interaction, these are all still descriptive
classifications, which makes them difficult to use in a generative and
design purpose.
They also do not describe ``softer'' and less goal-driven aspects of
interaction design such as aesthetics, user experience, and rewarding
interaction.

Below we discuss both the visual and the interactive aspects of
visualization design.

\subsection{Visual Aspects}

Textbooks on graphic design are an excellent source of information on
maximizing clarity and expressiveness in static (often printed) visual
representations.
Bertin~\cite{Bertin1967} discuss retinal variables for visual marks in
data visualization (later improved and extended by
Mackinlay~\cite{Mackinlay1986} and again by Card et
al.~\cite{Card1997}).
Furthermore, Edward Tufte's books~\cite{Tufte1983, Tufte1990,
  Tufte1997, Tufte2006} are key resources for visual design of data
displays, and have influenced many InfoVis systems and papers through
the years.
Following this tradition, Stephen Few's books give clear and concrete
design guidelines for how to design graphs and tables~\cite{Few2004}
as well as information dashboards~\cite{Few2006}.

Visualization textbooks are typically even more relevant because they
focus on interactive visualization applications.
Colin Ware's books~\cite{Ware2004, Ware2008} on perception and
cognitive aspects of visualization provide valuable background on the
psychology of visual thinking for interactive visualizations.
Robert Spence takes a design and interaction-oriented approach to
information visualization in his textbook~\cite{Spence2007}.
A recent book, \textit{Beautiful Data}~\cite{Segaran2009}, gives a
hands-on approach to visualization design through a set of case
studies involving real datasets collected from the web.
Most recently, Ward et al.~\cite{Ward2010} devote a full chapter to
step-by-step guidelines on how to design effective visualizations.

\subsection{Interactive Aspects}

While visual aspects are important for fluid interaction, it is
clearly the interactive aspects that are central for effective
visualization interaction design.
Unfortunately, interaction is not discussed at all in graphic design,
and even visualization textbooks tend to downplay this angle.

One exception is Robert Spence's book~\cite{Spence2007}, which takes
interaction design as its starting point for the study of information
visualization. 
Many of the visualization techniques presented in the book include a
discussion on key interaction design aspects associated with the
visual representation.
Another exception is Stephen Few's newest book~\cite{Few2009}, which
incorporates a chapter on analytical interaction where Few gives
recommendations for how visualization software should best support the
analytical discourse.
Few even uses the word ``fluid'' to describe a desirable feature of
the interactive exploration process (~\cite{Few2009}, pp.\ 82).
Our work in this paper builds on these existing efforts, but we
formalize the concept of flow and fluidity further.

The recent paper by Pike et al.~\cite{Pike2009} on the ``science of
interaction'' is highly relevant to our work.
Pike makes a case for the role of interaction in visualization and
visual analytics, and emphasizes themes such as interaction design,
user experiences, and best practices for interactive tools---the very
same themes we promote here.
The paper ends with seven broad areas for future research, but does
not take the practical design approach that our paper does.

Given the dearth of interaction design in InfoVis research, it is
fortunate that there exists much HCI research that is highly relevant
to the concepts of fluidity and flow in visualization.
We draw on a wide variety of disciplines---HCI and interaction design
included---when we define fluid interaction in the next section and we
will thus cite and discuss these sources below.
However, the most influential work includes the direct manipulation
paradigm~\cite{Shneiderman1983}, Norman's gulfs of execution and
evaluation~\cite{Norman1986}, the instrumental interaction
model~\cite{Beaudouin-Lafon2000, Beaudouin-Lafon2004},
tangible~\cite{Ishii1997} and embodied interaction~\cite{Dourish2001},
and, most recently, the reality-based interaction
framework~\cite{Jacob2008}.

In the next section, we show how this rich collection of related work,
theories, and frameworks can be tied together into the concept of
fluid interaction and how it can be applied to information
visualization.

\section{Fluid Interaction for Information Visualization}
\label{sec:definition}

As can be seen from the above literature survey, there exists very
little work on the topic of \textit{fluid interaction} for information
visualization.
Part of the reason for this is that notions of
flow~\cite{Csikszentmihalyi1991} and fluidity are very elusive and
difficult to pin down~\cite{Bederson2004}.
In this section, we will make inroads towards an operational
definition of this concept.
In the next section, we will describe a set of existing applications
that fulfill this definition: we call them InfoVis exemplars.
Finally, in the section following that, we synthesize the
characteristic properties collected from these examplars into general
guidelines for how to design fluid InfoVis applications.

\subsection{Operational Definition}

As mentioned above, fluid interaction is an elusive concept that is
not easily amenable to a theoretical definition.
Instead, we present an \textit{operational definition} of the
properties of fluidity that draws from a plethora of sources,
including HCI design, the concept of flow~\cite{Csikszentmihalyi1991},
embodiment~\cite{Dourish2001}, immersion, and natural interaction.
As a starting point, Merriam-Webster defines fluidity as
follows:

\begin{description}
\item[fluidity, n.:] 1. the quality or state of being fluid; 2. the
  physical property of a substance that enables it to flow.
\end{description}

Following this general definition, a \textit{fluid interface} for
information visualization is characterized by one or several of the
following properties:

\begin{itemize}

\item\textbf{Promotes flow:} The interaction should be designed to
  promote staying in the flow.
  ``Flow''~\cite{Csikszentmihalyi1991} is defined as a mental state
  of total immersion in an activity where the challenges of the
  activity and the skills of the participant are perfectly balanced,
  leading to high focus, involvement, and rewarding outcomes.
  Bederson~\cite{Bederson2004} previously proposed an interaction
  design philosophy based on helping the user stay in the flow,
  emphasizing five characteristics for user interface design.
  In the below list, we take a somewhat broader view of
  Csikszentmihalyi's factors for flow that may influence the concept
  of fluid interaction:
  
  \begin{itemize}
  \item\textit{Balanced challenge:} the skill required by the activity
    and the user's skill level should be matched;
  \item\textit{Concentration:} the activity should allow for a high
    degree of focus on a limited field of attention;
  \item\textit{Loss of self-consciousness:} the user should be able to
    merge action and awareness;
  \item\textit{Transformation of time:} enable users to ``lose
    themselves'' in the activity, essentially losing track of time;
  \item\textit{Prompt feedback:} users should be immediately informed
    of progress towards their goals; 
  \item\textit{Sense of control:} ensure users feel in control over
    the activity so that they can truly affect the outcome; and
  \item\textit{Intrinsically rewarding:} the activity should have a
    tangible reward in and of itself.
  \end{itemize}
  
\item\textbf{Supports direct manipulation:} The direct manipulation
  paradigm~\cite{Shneiderman1983} (further extended by the
  instrumental interaction model~\cite{Beaudouin-Lafon2000,
    Beaudouin-Lafon2004}) promotes an explicit method of interacting
  with computers by directly interacting with the domain objects
  themselves, thereby minimizing the indirection in the interface.
  The paradigm is based on four main principles:

  \begin{itemize}
  \item Continuous representation of the object of interest;
  \item Physical actions instead of complex syntax;
  \item Rapid, incremental, and reversible operations whose impact on
    the object of interest is immediately visible; and
  \item Layered or spiral approach to learning that permits usage with
    minimal knowledge.
  \end{itemize}

\item\textbf{Minimizes the gulfs of action:} According to usability
  expert Donald Norman, the challenge of interacting with any system,
  physical or virtual, can be described in terms of two
  \textit{gulfs}~\cite{Norman1986}:
  
  \begin{itemize}
  \item\textit{Gulf of Evaluation}: The difference between the
    system's state and the user's perception of that state.
  \item\textit{Gulf of Execution}: The difference between the
    allowable actions of a system and the user's intentions for using
    the system.
  \end{itemize}

\end{itemize}

\subsection{Towards a Cognitive Account of Fluid Interaction} 

From a cognitive perspective, the aforementioned desired properties of
a \textit{fluid interface} for InfoVis share some important
commonalities.
Understanding these commonalities and identifying an underlying common
concept could help to better understand what creates and what hampers
fluidity in InfoVis.
In the following, we suggest such a common concept which is deduced
from existing cognitive models.
This first step towards a cognitive account of fluidity is still far
from being a comprehensive theoretical model.
However, viewing visualization design through this new lens could help
to analyze existing information visualizations and to inform and
refine future models.

In our view, a basic requirement for fluidity and the underlying
concept behind it is the users' feeling of direct participation and
embodiment in the interface.
Fluid interfaces for InfoVis must make the users feel that are able to
directly ``touch'' and manipulate the visualization instead of
indirectly conversing with a user interface.
Users should get a feeling of immersion, first-personness and direct
engagement with the objects and the visualizations that concern them.

This phenomena was first described by Hutchins et
al.~\cite{Hutchins1985} in their cognitive account of direct
manipulation.
They differentiated between two major metaphors for the nature of
human-computer interaction, a \textit{conversation metaphor} and a
\textit{model-world metaphor}: in the former, the interface serves as
a language for interacting with the world, whereas in the latter, the
interface itself is the world which the user can manipulate.
For Hutchins et al., model-world interfaces create a feeling of
directness and direct manipulation by minimizing the \textit{gulfs of
  evaluation} and \textit{execution}~\cite{Norman1986} and thereby
using much less of the users' cognitive resources.

We believe, that InfoVis---and in particular \textit{fluid}
InfoVis---should follow this model-world metaphor.
This is also in line with the growing importance of theories of
\textit{embodied cognition} in cognitive science and \textit{embodied
  interaction} in human-computer interaction~\cite{Dourish2001}.
These embodied views emphasize that our cognitive abilities are
specifically designed to reason, act, and move in our natural physical
and social world.
In other words, our entire way of thinking and our perception are
optimized for these real-world tasks and cannot be separated from our
physical and social existence.
These physical and social skills are far more constituent of what we
consider as human cognition than the disembodied and formal processing
of symbols when conversing with a computer by clicking through
labelled buttons, menus, hyperlinks or forms.
Thus, we are wasting a great deal of our true skills when using
computing technology without directly acting in metaphorical
(model-)worlds.

This insight has lead the field of HCI to a new generation of user
interfaces based on \textit{reality-based
  interaction}~\cite{Jacob2008}.
These interfaces use modalities such as body tracking, bi-manual
multi-touch interaction, or tangible objects to further reduce the
users' \textit{gulf of execution} and to draw strength by employing
themes of reality such as \textit{body}, \textit{environment} and
\textit{social skills \& awareness}~\cite{Jacob2008}.
Thus they transfer an even stronger form of direct manipulation from
desktop to post-desktop computing.

In conclusion, given our innate cognitive abilities, creating
model-world interfaces into which users can immerse themselves and in
which objects of concern become virtually or even physically tangible
can help to achieve Csikszentmihalyi's factors of flow.
Directly manipulating the objects in the model-world with a greater
set of motor skills (e.g., directly dragging an object with
multi-touch or mouse instead of pushing keys or buttons to that
effect) mediates a \textit{sense of control} and the model-worlds
provide the desired \textit{prompt feedback}.
They use less cognitive resources and thus enable
\textit{concentration} on the task instead of concentration on
handling the user interface.
Less usage of cognitive resources can also help to achieve a greater
design space for a more \textit{balanced challenge}.
Furthermore, good direct manipulation interfaces are in many respects
similar to computer games~\cite{Shneiderman1983}, and thus could lead
to the \textit{loss of self-consciousness} and \textit{transformation
  of time}.

\subsection{Utility of Fluid Interaction}

Our argument so far has assumed that fluid interaction is a desirable
attribute in visualization design.
However, it is true that fluidity is not a necessary condition for any
given visualization application, and we can even go so far as to say
that there probably exist numerous very successful visualization and
analysis applications that score low on what we would call ``core''
fluidity properties. 
For example, the statistical package R
(\url{http://wwwr-project.org/}) has a command-based user interface
with few visual components (i.e., a conversation interface), but is
widespread and highly successful in many communities (we should note
that R certainly does promote flow for expert users, while other
fluidity properties are somewhat neglected).

On the other hand, interaction is the catalyst for the interplay
between the data and the user, and is an essential part of visual
exploration~\cite{Dix1998, Norman1994, Yi2007}.
Thus it follows that achieving the ``optimal experience''
(characterized as flow~\cite{Csikszentmihalyi1991}) while interacting
with a visualization application will cause the user to perform
better~\cite{Bederson2004}.
In other words, if we improve the fluidity of a visualization
application, user performance would directly benefit.
This is also the message of this paper: showing how interaction design
can be used to make visualization applications, existing and novel
ones alike, more effective.
For example, RStudio (\url{http://www.rstudio.org/}) is a new
graphical IDE for R that likely will help novice users overcome the
steep learning curve of the R system.

Many authors argue in favor of streamlining the interactive process in
this way.
Pike et al.~\cite{Pike2009} emphasize the need for natural and
seamless interaction methods in support of discovery.
Although they note that disruptions in the analytical discourse are
inevitable and, in fact, often useful, their use of the word ``fluid''
is different than ours, and their argument still seems to advocate the
user staying in the flow of a visualization tool during analysis.
Bederson~\cite{Bederson2004} use interruptions as a reasoning tool
when arguing for maintaining flow in interactive applications, noting
that literal or conceptual interruptions can have large impact on user
productivity.
While sensemaking literature~\cite{Russell1993} typically discusses
interaction with information at a conceptual level, many of Pirolli
and Card's leverage points~\cite{Pirolli2005} directly or indirectly
involve improving the cost structure of analysis through tool
innovation.

Given the above operational definition, cognitive account, and utility
for fluidity, we are now ready to study concrete examples that embody
these concepts.

\section{InfoVis Exemplars}
\label{sec:exemplars}

In this section, we review six InfoVis exemplars that we feel exhibit
the fluid interaction properties discussed earlier in this article.
Table~\ref{tab:overview} gives a summary of the exemplars based on
domain (the primary community the system targets), audience (intended
users), task (the primary task and intention with the system), and
properties (the primary properties of fluidity that the system
exhibits).
We describe each of these systems in detail below.

Of course, any choice of a mere six examples from the rich array of
excellent visualization systems is highly subjective and could even be
seen as somewhat arbitrary.
We used the following criteria when identifying the exemplars:

\begin{itemize}

\item\textbf{Diversity:} We wanted our selection to reflect the
  practice of information visualization in both academic and design
  communities, as well as on the web, in movies, and in physical
  locations such as museums. 

\item\textbf{Illustrative:} Our choice was influenced by a desire to
  illustrate \textit{different}---seemingly disparate---aspects of
  each system that the concept of fluidity could help explain.

\item\textbf{Availability:} We chose only work that that was directly
  available (through demonstrations or videos) and amply described in
  the literature (for academic work) or in the blogosphere or popular
  press (for non-academic work).
  In other words, we relied on peer-review (academic work) or public
  acceptance (non-academic work).
  
\item\textbf{Depth instead of breadth:} This paper is not a
  comprehensive survey of best-in-class visualizations, and thus our
  focus was on deep analysis of each exemplar instead of a large and
  broad enumeration of \textbf{all} InfoVis systems that incorporate
  fluid interaction.

\end{itemize}

Other factors beyond our control naturally also played a role in this
selection process, including the authors' own biases, personal tastes,
and familiarity with the literature.
However, regardless of the sparse sampling of the design space, we are
convinced that the characteristic properties that we extract from each
exemplar together are broadly applicable for fluid interaction in
general.


\begin{table}[tbh]
  \centering
  \begin{tabular}{lllll}
    \hline
    \textbf{Exemplar} & \textbf{Domain} & \textbf{Audience} &
    \textbf{Task} & \textbf{Properties}\\
    \hline

    Facet-Streams~\cite{Jetter2011} & academic & groups &
    collaborative search & direct, embodied, minimizes gulfs\\

    BabyNameVoyager~\cite{Wattenberg2005} & academic/design & web &
    exploration & direct, prompt feedback, rewarding, control\\

    Scatter/GraphDice~\cite{Bezerianos2010, Elmqvist2008a} & academic &
    analyst & exploration & direct, prompt feedback, minimizes gulfs\\

    M{\ae}ve~\cite{Maeve, Nagel2009} & museum & casual & browsing &
    embodied, rewarding, aesthetic\\

    We Feel Fine~\cite{Kamvar2009} & design & web & social navigation
    & rewarding, aesthetic, minimal knowledge\\
    
    Iron Man 2~\cite{IronMan2} & movie & casual & immersion &
    embodied, feedback, aesthetic\\
    
    \hline

  \end{tabular}
  \caption{Overview of the six InfoVis exemplars reviewed in this
    paper with informal classifications on the domain, intended
    audience, main task, and primary fluid properties of each system.}
  \label{tab:overview}
\end{table}

\subsection{Facet-Streams}

Facet-Streams~\cite{Jetter2011} is a system for co-located
collaborative product search by multiple users around a tabletop
(Figure~\ref{fig:facet-streams1}).
It supports small groups during decision-making and negotiation by
enabling a faceted exploration of a product catalog, e.g.\ a catalog of
hotels for a family's vacation.
Queries can be formulated by each participant by putting small glass
discs as query tokens on the tabletop and assigning the desired data
field and value ranges to them (Figure~\ref{fig:facet-streams2}).  
These tokens can then be visually linked to form a directed graph that
serves as a visual filter/flow representation \cite{Young1993} of
faceted Boolean search.
All products from the catalog flow along the edges of the network and
are filtered by the nodes they pass.
Logical AND and OR can be expressed without symbolic notations or
query languages, simply by connecting nodes or letting edges flow
together. 
Query results can be inspected by the users by touching an edge to
reveal the products that flow therein.

\begin{figure}[htb]
  \centering
  \resizebox{!}{4.2cm}{\includegraphics{figures/facet-streams1.jpg}}
  \resizebox{!}{4.2cm}{\includegraphics{figures/facet-streams2.jpg}}
  \caption{The Facet-Streams system for co-located collaborative
    search on tabletops (left). 
    Glass tokens form a tangible filter/flow representation of faceted
    Boolean search (right).}
  \label{fig:facet-streams1}
\end{figure}

\begin{figure}[htb]
  \centering
  \resizebox{!}{5.5cm}{\includegraphics{figures/facet-streams3.jpg}}
  \resizebox{!}{5.5cm}{\includegraphics{figures/facet-streams4.jpg}}
  \caption{Assigning the desired data field (left) and value range
    (right) to a query token using touch input.}
  \label{fig:facet-streams2}
\end{figure}

Through tangible and touch input, Facet-Streams exploits a greater
range of the users' real-world motor skills than normal desktop
interfaces.
The number and spatial layout of nodes can be altered by familiar
physical manipulations, similar to placing, lifting or moving the
pieces of a board game like checkers.
The topology of the network can be changed by touch interaction,
e.g.\ by dragging new connections between nodes with the fingers or by
cutting them with a crossing out gesture.
The selected data fields and value ranges of a node can be changed by
touching and sliding the finger over the token's field and value dials
(Figure~\ref{fig:facet-streams2}).
This also enables users to develop more advanced techniques,
e.g.\ bi-manual selection of value ranges during which one hand
rotates the glass token and its attached dial while a finger of the
other hand selects the segments of the dial that is rotating below.
Thereby every kind of tangible and touch input into the system leads
to immediate visual feedback.
For the users, this creates the illusion of direct physical
interaction with the visual representation.

The benefit of this design is a low viscosity of the query's visual
representation, i.e., a ``low resistance to change'' in the
interface~\cite{Jetter2011}.
It enables users to rapidly modify it according to their individual or
shared goals.
During initial search phases, each group member can formulate and
explore their own criteria individually during phases of
loosely-coupled parallel work.
These personal query networks can then be effortlessly combined into a
larger group network for collective reviewing during phases of
tightly-coupled collaboration.
However, the query's network can easily be dissolved into smaller
parts again, e.g.\ for returning to parallel work or to separate the
satisfactory parts from those that need further refinement.
Therefore, low viscosity also gives the necessary flexibility to
support different working styles or different collaborative phases.

In summary, we can extract the following characteristic properties of
fluidity from Facet-Streams:

\begin{itemize}
\item\textit{Tangible and touch interaction} with immediate visual
  feedback creates the illusion of physically interacting with the
  visual representation;
\item This results in a \textit{low viscosity} of the visual
  representation---i.e., a low resistance to change---that enables
  users to rapidly modify it according to their individual or shared
  goals; and
\item\textit{Flexibility in working styles} for the group of users by
  enabling smooth changes between loosely-coupled parallel work and
  tightly-coupled collaboration.
\end{itemize}

\subsection{BabyNameVoyager}

The BabyNameVoyager~\cite{Wattenberg2005} is a web-based
visualization tool (accessible from
\url{http://www.babynamewizard.com/name-voyager}) for interactive
representation and analysis of historical trends in baby naming.
Figure~\ref{fig:babyname} shows a screenshot of the application where
a subset of names have been filtered out. 
The data is represented as a set of stacked graphs~\cite{Byron2008},
where the horizontal axis represents the time, and the vertical axis
the amount of children that were given a certain name in the
corresponding period.  
Initially, all names in the dataset are displayed as narrow colored
overlaying threads that are sorted alphabetically.
Filtering---the most important interaction in this tool---can be
achieved in two different ways that both exhibit a high degree of
fluidity: by interactively browsing the data, or by issuing a partial
textual query.

\begin{figure}[htb]
  \centering
  \resizebox{0.6\textwidth}{!}{\includegraphics{figures/babyname}}
  \caption{A screenshot of Baby Name Voyager showing names starting
    with ``Mari''.}
  \label{fig:babyname}
\end{figure}

The user can browse the data by moving the mouse over the name
segments.
Clicking a segment will select the corresponding name and expand it,
while filtering out the other stacks in a gentle animation.
The smooth transitions between states provide continuity between the
information presented in different states.
Moreover, the presence of a direct interface with the visualization
suggests similarities to other real-world exploration tasks (e.g.,
browsing a drawer of ordered folders; clicking can be viewed as the
selection and smooth opening of a folder to reach the actual
information).

Textual queries are issued by typing a name, or parts of a name, into
a text box. 
The filtering is immediate, as each key stroke generates a new
intermediary name and a stricter filtering rule, causing the visual
display to update as the user is typing.
Again, the interaction mimics widely known and used actions from our
everyday life, which suggests a coupling between fluidity and
real-world metaphors.

In summary, we can extract the following characteristic properties of
fluid interaction from the BabyNameVoyager:

\begin{itemize}
\item\textit{Smooth animated transitions} between visualization
  states;
\item\textit{Minimalistic interface} using direct
  manipulation~\cite{Shneiderman1983} (click-to-query) or an
  integrated query box (textual queries);
\item\textit{Immediate visual feedback} for not only final queries,
  but also intermediate ones; and
\item\textit{Aesthetic visual design} that does not sacrifice
  correctness.
\end{itemize}

\subsection{ScatterDice and GraphDice}

ScatterDice~\cite{Elmqvist2008a} (2008) and
GraphDice~\cite{Bezerianos2010} (2010) are visualization tools for
interactive visual exploration of multidimensional tables and
multivariate graphs, respectively.
Figure~\ref{fig:graphdice} shows a screenshot of the GraphDice
interface (the ScatterDice interface is similar, but uses scatterplots
instead of node-link diagrams).
As can be seen from the image, the interface is dominated by the
current plot (a 2D scatterplot for ScatterDice and a node-link graph
with attribute-based layout for GraphDice), as well as a plot matrix
that shows all of the dimensions in the dataset.
Users can change the current plot by navigating in the plot matrix;
the transition from one position to another is communicated using an
animated 3D rotation.

\begin{figure}[htb]
  \centering
  \resizebox{\textwidth}{!}{\includegraphics{figures/graphdice}}
  \caption{The GraphDice~\cite{Bezerianos2010} multivariate graph
    visualization tool showing the current plot (right), the plot
    matrix (top center) and the query layer box (bottom center).
    The integrated spreadsheet-style table (left) shows details on
    demand.
    A FaST-slider~\cite{McGuffin2002} (active in the current plot)
    allows for quick navigation between data dimensions.}
  \label{fig:graphdice}
\end{figure}

This navigation method---called \textit{scatterplot matrix
  navigation}---is an example of fluid interaction because (1) the
transition is smoothly animated with no discontinuities, and (2) it
builds on a cohesive and consistent model for the visual
representation that is closely integrated with an interaction technique
for navigation in data (and not just geometric) space.
In this way, scatterplot matrix navigation minimizes both the gulfs of
evaluation and execution~\cite{Norman1986}.
The method is also integrated with another fluid interaction technique
called \textit{query sculpting} where users select data items in one
plot using a lasso or selection box, and can refine their queries in
other plots as they navigate the data space.
As the plots change, the \textit{query hulls} representing each
selection animate smoothly as well to maintain the user's mental model
of the exploration.

In summary, we can extract the following characteristic properties of
fluidity from the ScatterDice and GraphDice tools:

\begin{itemize}
\item\textit{Smooth animated transitions} between visualization
  states;
\item\textit{Minimalistic interface} using direct
  manipulation~\cite{Shneiderman1983} (query sculpting);
\item\textit{Immediate visual feedback} for both query sculpting and
  navigation operations; and
\item\textit{Coherent conceptual model} that allows users to think and
  reason about the visual representation.
\end{itemize}

\subsection{M{\ae}ve}

M{\ae}ve~\cite{Maeve, Nagel2009} is an interactive multi-touch
tabletop application for a museum installation
(Figure~\ref{fig:maeve}) that employs a tangible interaction metaphor
to enable the manipulation of a network graph, which itself represents
the relationships between various architectural projects.
When a paper card of a specific project is placed on the tabletop,
its associated information structure appears, including media files,
keywords, and related projects.
The new card is then also visually connected to other cards that might
be present in the table, to highlight the various similarities between
the projects.

\begin{figure}[htb]
  \centering
  \resizebox{\textwidth}{!}{\includegraphics{figures/maeve-installation}}
  \caption{M{\ae}ve~\cite{Maeve, Nagel2009} installation at the Venice
    Biennale 2008, with a detail (inset) of the interactive tabletop
    surface.}
  \label{fig:maeve}
\end{figure}

According to Bardzell et al.~\cite{Bardzell2010}, ``It is easy to
imagine a traditional browser-based presentation [...] But when you
approach the same information through m{\ae}ve, something happens.''  
This ``something,'' according to Bardzell et al., is the ultimate
combination of form and content, and content and interface.
The result is that the visualization becomes an experience rather than
a (productive) tool.
Even more, sharing some characteristics with cinema, m{\ae}ve is set
up inside a fully controlled, darkened performance space, displaying
user interactions projected on the wall in the space.

In general, m{\ae}ve shows how the concepts of fluidity can transcend
beyond the traditional screen media.
Whereas the typical interface constraints are more apparent and
restricting when working on a multi-touch tabletop medium, there is
little that withholds the actual translation of these principles to
typical information visualization applications on classic screen
media.

M{\ae}ve exhibits the following fluid characteristics:

\begin{itemize}
\item\textit{Responsive and immediate} visual feedback to interaction;
\item\textit{Focus on experience} rather than productivity; and
\item\textit{Powerful and effective use of novel inputs and outputs}
  that encourages experimentation.
\end{itemize}

\subsection{We Feel Fine}

We Feel Fine~\cite{WeFeelFine}, developed by Jonathan Harris and Sep
Kamvar (\url{http://www.wefeelfine.org/}), is an interactive
exploration of contemporary human feelings.
Its data is based on over 12 millions blog posts starting with the
terms ``I feel...'' that have been published online since 2005.
Using a series of playful interfaces, these feelings can be searched
and sorted across a number of demographic slices, offering responses
to specific questions like: ``Do Europeans feel sad more often than
Americans?'' or ``Does rainy weather affect how we feel?''  

\begin{figure}[htb]
  \centering
  \resizebox{\textwidth}{!}{\includegraphics{figures/wefeelfine}}
  \caption{Sample interaction in the We Feel Fine application.}
  \label{fig:wefeelfine}
\end{figure}

The interface uses a kinetic metaphor of a continuously
self-organizing particle system, where each particle represents a
single feeling posted by a single individual.
The particles' properties, such as their color, size, shape, opacity,
indicate the nature of the feeling inside.
The particles move about randomly around the screen until requested to
self-organize in pictograms or along a number of data axes, thereby
plotting and expressing various patterns of human emotion.
The online application, which also is disseminated in a documentary
book~\cite{Kamvar2009}, provides an engaging view of predominantly
qualitative data that is inherently interesting and relevant to the
user.
The interaction is immediate and playful, where clicking (and even
just moving) the computer mouse becomes an overarching goal on its
own, with a visceral experience as a joyful reward.
While the content pushes the user's curiosity, the real user
experience is the interaction.
The interaction flow never really ends: there is always more to
select, discover and explore, immersing the user in a true information
experience. 
The aesthetics are compelling, with a consistent visual language and
well-executed details throughout (e.g.\ shivering particles).
Fluidity finds its summon in terms of how interactivity, visual design
and content complement and augment each other.

We extract the following characteristic properties of fluid
interaction from We Feel Fine:

\begin{itemize}
\item\textit{Immediate and playful} interaction;
\item\textit{Never-ending interaction flow} that encourages
  exploration; and
\item\textit{Holistic interaction, visual design, and content} that
  complement and reinforce each other.
\end{itemize}

\subsection{Interactive Holographics in Iron Man 2}

Since the rising popularity of computer interfaces, the movie and
television industry has embraced the persuasiveness of complex data
representations.
For the movie \textit{Iron Man 2}~\cite{IronMan2}, special effects
design firm Prologue (\url{http://www.prologue.com/}) designed a
collection of incredibly dense information dashboards and highly
responsive real-time 3D interfaces that demonstrate an impressive
futuristic view of intuitive depictions of complex data
(Figure~\ref{fig:ironman2}).

\begin{figure}[htb]
  \centering
  \resizebox{!}{5cm}{\includegraphics{figures/ironman2-1}}
  \resizebox{!}{5cm}{\includegraphics{figures/ironman2-2}}
  \caption{Tony Stark (Robert Downey, Jr.) interacting with the
    information-rich holographic displays in \textit{Iron Man 2}.}
  \label{fig:ironman2}
\end{figure}

The core of their vision builds upon the current knowledge about
tabletop interaction, and imagines a future wherein information can be
presented in a truly pervasive and directly manipulable way.
Hand gestures, finger snapping and pinches make information immediate
queryable, results can be pushed aside and even thrown, and the highly
contemporary design of the interfaces makes mundane task-oriented
actions fun to accomplish.
In terms of fluidity, Iron Man 2 showcases opportunities when screen
and interaction media physically disappear, and information immerses
the user in life-size, high-definition resolution.

While the interfaces shown in the movie might neglect some obvious
usability and usefulness constraints, they effectively demonstrate how
far the typical characteristics of fluidity can be pushed in terms of
immediateness, smoothness, and, in particular, the expressiveness of
data.
As with multi-touch technology, the persuasiveness of this future
vision will further inspire many research agendas in the immediate
future, and influence the way we will interact with information.
In fact, with the emergence of ever-increasing LCD display sizes and
the recent commercial releases of affordable controller-less
interaction devices such as the Microsoft Kinect, this future might
not be that far off.

The interactive holographics in Iron Man 2 exhibits the following
characteristics:

\begin{itemize}
\item\textit{Reality-based interaction}~\cite{Jacob2008} that
  transfers natural interactions such as gestures and body language to
  the interaction with data and visual displays; and
\item\textit{Immediate visual feedback} that responds instantly to
  user interaction.
\end{itemize}

\section{Design Guidelines for Fluidity}
\label{sec:implications}

Drawing on our observations and insights from the above review of
InfoVis design exemplars, we here present a set of design guidelines
for how to design and build effective information visualizations that
support fluid interaction.
While this list is far from exhaustive, our hope is it will serve as a
common ground for other researchers and designers to improve and
extend upon in the future.

\begin{itemize}

\item\textbf{DG1:} \textit{Use smooth animated transitions between
  states.}
  Animated transitions help the user maintain an accurate mental model
  of the system's current state.
  Avoid abrupt mode switches because they are potentially disorienting
  and may break the user's flow.
  However, animation is a very strong visual
  variable~\cite{Tversky2002} and designers should be careful not to
  take this design guideline too far.
  There exists a rich literature on this topic; e.g.~\cite{Hudson1993,
    Heer2007a, Robertson2008, Dragicevic2011}.
  
\item\textbf{DG2:} \textit{Provide immediate visual feedback on
  interaction.}
  Do this for \textit{every} key press or mouse motion, not just
  ``major'' events ones like mouse clicks and the Enter key.
  In particular, this means that the visualization must be able to
  respond in real-time; if this is not possible, consider precomputing
  or simplifying some computation so that it becomes possible.

\item\textbf{DG3:} \textit{Minimize indirection in the interface.} 
  If at all possible, use direct manipulation~\cite{Shneiderman1983}
  so that interaction operations (filtering, selection,
  details-on-demand) are integrated in the visual representation.
  In particular, avoid control panels that are separated from the
  visualization, or, if this is not possible, only put seldomly used
  controls there.

\item\textbf{DG4:} \textit{Integrate user interface components in the
  visual representation.}
  If you cannot use direct manipulation and must use traditional
  interface components, like text fields, sliders, or buttons, try to
  make them a seamless, nearly embodied, part of the visualization.
  The interface should be nearly ``invisible.''

\item\textbf{DG5:} \textit{Reward interaction.} 
  Users should be encouraged to interact with a visualization so that
  the dialogue between user and system is initiated and maintained.
  Visually indicate how the user can interact with the visual
  representation, and when they do, give them a visceral reward.
  These rewards are effects that trigger a positive user experience
  without actually having a function in terms of visual exploration or
  visual communication; instead, their purpose is to keep the user
  stimulated while exploring.
  Examples of rewards include animations, sounds, and pretty graphics.
  
\item\textbf{DG6:} \textit{Ensure that interaction never ``ends.''}
  The user should never reach a dead end where they can no longer
  proceed; it should always be possible to continue exploring the
  data.
  The system should be robust so that it allows all interactions
  without fear of crashing, long response times, or irreversible
  operations.

\item\textbf{DG7:} \textit{Reinforce a clear conceptual model.}
  The user should always have a clear idea of the state of the
  visualization and all interactions should be designed to reinforce
  this model.
  Operations should be reversible, allowing the user to return to a
  previous state.
  For visualizations using coordinated multiple views, connections
  between views should be clear and visible to the user.

\item\textbf{DG8:} \textit{Avoid explicit mode changes.} 
  Instead of introducing different modes, integrate all operations in
  the same mode.
  This includes avoiding both drastic visual changes and drastic
  interaction modality changes.
  Mode changes may break the user's flow. 

\end{itemize}

\section{Vision and Research Directions}
\label{sec:vision}

Interaction has a central role in visualization despite typically
receiving much less emphasis than visual aspects~\cite{Yi2007}: it is
the catalyst for the user's dialogue with the data and, ultimately,
for attaining insight and understanding.
This dialogue between the user and the system is what sets our field
apart from statistical data graphics and infographics, and it is clear
this is where the true potential of visualization comes to the fore.
A research agenda focusing on fluid interaction for information
visualization will help fulfil this potential.
However, we have barely scratched the surface of this exciting new
line of research, and much work remains to be done.
In the following subsections we outline our vision for future research
directions in this area.

\subsection{InfoVis Exemplar Repository}

In this paper, we review six different tools that serve as exemplars
of fluid interaction for information visualization, three from the
research community and three from the design and web community.
Naturally, there are many more tools that could have been included in
this collection, and our selection was mainly based on space
constraints and the familiarity of the authors with different tools.
Furthermore, we also do not mean to imply that the six tools reviewed
here are paragons of perfection, but rather that they possess the kind
of fluid interaction features that we wanted to highlight.
Therefore, we hope that the reader will accept this review in the
spirit it was collected in: a set of excellent visualizations that
together help define the concept of fluidity in information
visualization.

Having said that, a worthwhile future research direction may be to
continue collecting these InfoVis exemplars into repositories for
students, researchers, and designers alike to use as sources of
inspiration and reference for their own projects.
Such a repository should probably be curated (like the examples in
this paper) and include both commentaries and reviews by people other
than the creators themselves to highlight the strengths (and
weaknesses) of each tool.

Exemplar repositories of this kind would be similar to Edward Tufte's
books, which essentially are catalogues of good graphic design.
However, while the examples in Tufte's books are curated by the author
himself, a larger question would be who should curate these InfoVis
exemplars?
Should it be single expert, or groups of experts?
Should we have a single exemplar repository, or several?
There may be a need to split these examples into different categories
for academic, commercial, design and art---as we have seen in this
paper, the criteria for excellence vary widely between these different
communities.

Fortunately, there already exists a set of blogs and online
collections that already serve as exemplar repositories in this
regard, and with the blog authors as the informal curators.
For example, the \textit{information
  aesthetics} (\url{http://infosthetics.com/}) weblog, created
and maintained by one of the authors of this paper, ``explores the
symbiotic relationship between creative design and the field of
information visualization.''
Many of the design exemplars discussed in this paper are derived from
entries posted on the infosthetics website.
Other similar resources include Manuel Lima's
\textit{VisualComplexity.com}
({\url{http://www.visualcomplexity.com/}), Michael Friendly's
  \textit{Gallery of Data Visualization}
  (\url{http://www.math.yorku.ca/SCS/Gallery/}, which also includes
  counter examples of bad visual design), Robert Kosara's
  \textit{EagerEyes} (\url{http://eagereyes.org/}), and the Potsdam
  Information Design Patterns website
  (\url{http://infodesignpatterns.com/}).

\subsection{Visualization Design Patterns}

While our above design guidelines are short and practical guidelines
on building effective and fluid information visualizations, an
ultimate goal of this research may be to formalize the concept of
\textit{visualization design patterns} that build on the idea of
``design patterns'' that was originally devised for architecture by
Christopher Alexander in 1977~\cite{Alexander1977}, but which has
since been embraced by the software engineering
community~\cite{Gamma1995}.
According to Alexander, a design pattern can be described as follows:

\begin{quotation}
  ``Each pattern describes a problem which occurs over and over again
  in our environment, and then describes the core of the solution to
  that problem, in such a way that you can use this solution a million
  times over, without ever doing it the same way twice.'' ---
  Christopher Alexander, \textit{A Pattern Language}
  (1977)~\cite{Alexander1977}.
\end{quotation}

It is important to note the distinction between \textit{visualization
  design patterns} and the \textit{software design patterns for
  visualization} introduced by Heer and Agrawala~\cite{Heer2006}; the
latter deals with pure software engineering strategies for building
visualization software, whereas our proposed visualization design
patterns focus on high-level design aspects of the visual
representation, interaction, and user experience of the visualization
itself.
We think that there is a space and a need for defining this kind of
pattern language (or at least a pattern catalogue) for information
visualization in the same way as for the architecture and software
engineering fields, thus standardizing terminology, perpetuating best
practices, and improving visualization quality in general.

Adopting a pattern mindset is also advantageous because it allows us
to define the notion of an \textit{antipattern}~\cite{Koenig1995}:
standard solutions to common problems (in whatever field this is
applied to) that simply do not work (or work poorly).
Examples of such antipatterns may be the infamous rainbow color scale
(which both is not aesthetically pleasing according to most observers,
and more importantly has perceptual limitations~\cite{Munzner2008}),
serious and distracting overuse of animation~\cite{Tversky2002}, and
overly complex visualizations that hide, rather than expose, important
connections, or which show outright false relations.
In general, counter examples are instructive in teaching people what
\textit{not} to do---examples include Huff's classic \textit{How to
  Lie with Statistics}~\cite{Huff1954}, as well as Michael Friendly's
gallery (discussed above), which includes not only the best but also
the worst data visualizations on the Internet.
Generalizing these to actual antipatterns would be beneficial towards
increasing visualization quality overall.

\subsection{Towards Visualization Criticism}

Both education and professional practice in mature design
disciplines---such as architecture, industrial design, and graphic
design---incorporate the notion of \textit{criticism} as an integral
component of the design process.
This practice is slowly starting to be adopted for HCI and interaction
design as \textit{interaction criticism}~\cite{Bardzell2009,
  Bardzell2010}.
The concept of ``expert review'' has long been an established
evaluation methodology in HCI, but these new advances consider
aesthetics in addition to function and incorporate also the opinion of
practioners and users as part of the process.

Perhaps a similar notion of \textit{visualization criticism} should be
applied to our own field.
In fact, a recent survey~\cite{Kerren2008} shows that one of the most
common practical exercises in existing information visualization
courses is to critique existing InfoVis tools.
Furthermore, expert reviews have begun to gain traction in the
visualization domain as well~\cite{Tory2005}.
For the aesthetics component, the model proposed by Lau and Vande
Moere~\cite{Lau2007} goes beyond subjective judgments to analyze the
artistic influence and aesthetic engagement of a visualization.
However, all of these approaches tend to focus on the graphical
aspects of a visualization rather than the interactive aspects.

At the core of this paper lies the message that representation and
interaction deserve equal treatment in visualization research.
Therefore, we think that a worthwhile avenue for future work would be
to combine all of the above approaches together with interaction
criticism in an effort to identify visualization criticism as a
skilled practice in its own right, one that focuses on the interplay
between visuals and interaction along the lines of the discussions in
this paper.
Not only would this provide a method for curating visualization
exemplars (see above) as well as for educating the next generation of
visualization designers and researchers, but it would also bring
interactive---and not just visual---aspects of visualization tools to
the fore.

\section*{Acknowledgments}

We thank all of the participants of the Interaction in Information
Visualization discussion group at Dagstuhl Seminar 10241 (June 2010)
on ``Information Visualization'' for the initial discussions that led
to this article.

% ----------------------------------------------------------------------
% Bibliography
% ----------------------------------------------------------------------
\bibliographystyle{plain}
\bibliography{fluidity}

\end{document}
